Multi-path data retrieval from redundant array

ABSTRACT

An optimum pathway to data stored on a data storage system having N storage devices and more than N pathways is determined in response to a read request for the data. A sorter separates the read request into an appropriate segment size for sending to the storage devices of the data storage system. An assigner generates the set of read permutations satisfying the read request. A read permutation is selected based on a metric. A collector receives the requested data from the N storage devices in response to the selected read permutation being sent to the storage devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of patent application of U.S.patent application Ser. No. 12/016,597, filed Jan. 18, 2008, invented bySteven J. Hetzler et al., entitled “Multipath Data Retrieval FromRedundant Array, which is a continuation patent application of U.S.patent application Ser. No. 10/619,633, filed Jul. 14, 2003, invented bySteven J. Hetzler et al., entitled “Multipath Data Retrieval FromRedundant Array, the disclosures of each are incorporated by referenceherein. Additionally, the present application is related to patentapplication Ser. No. 10/619,641, entitled “Anamorphic Codes”, patentapplication Ser. No. 10/619,649, entitled “Autonomic Parity Exchange,”now U.S. Pat. No. 7,281,177 B2 to Hetzler et al., issued Oct. 9, 2007,and patent application Ser. No. 10/619,648, entitled “RAID 3+3,” nowU.S. Pat. No. 7,254,754 B2 to Hetzler et al., issued Aug. 7, 2007, eachco-pending, co-assigned and each filed Jul. 14, 2003, and the disclosureof each incorporated by reference herein. The present application isalso related to co-pending and co-assigned patent application Ser. No.10/600,593, which has been announced to be U.S. Pat. No. 7,350,126 toWinograd et al. on Mar. 25, 2008, the disclosure of which is alsoincorporated by reference herein.

BACKGROUND

1. Field

The subject matter disclosed herein relates to the field of data storagesystems. In particular, the subject matter disclosed herein relates asystem and a method for determining a best pathway to requested databased on a metric, thereby improving device failure protection of anarray of storage devices.

2. Description of the Related Art

A conventional array of storage devices typically has sufficientredundancy so that when a storage device fails, information contained inthe failed storage device can be reconstructed from the remainingstorage devices. See, for example, U.S. Pat. No. 5,579,475 to M. M.Blaum et al., entitled “Method and Means for Encoding and Rebuilding theData Contents of Up to Two Unavailable DASDs in a DASD Array UsingSimple Non-Recursive Diagonal and Row-Parity,” which discloses theoperation of an array having distance D=3. See also, N. K. Ouchi,“Two-Level DASD Failure Recovery Method,” IBM Technical DisclosureBulletin Vol. 36, 3 Mar. 1993, discloses the operation required forreconstructing data from an array with failures and having distance D=3.

Redundancy may also be used for improving performance. See, for example,E. J Schwabe et al., “Evaluating Approximately BalancedParity-Declustered Data Layouts for Disk Arrays,” ACM0-89791-813-4/96/05 1996, which disclose data layouts for efficientpositioning of redundant information for performance. See also G. A.Alvarez et al., who, in “Tolerating Multiple Failures in RAIDArchitectures,” ACM 0-89791-901-7/97/0006 1997, disclose properties andconstruction of a general multiple-parity array using 8-bit finitefields, and L. Xu and J. Bruck, who, in “Improving the Performance ofData Servers Using Array Codes,” Paradise ETR027 (CalTech) 1998,describe the use of a maximum distance separation (MDS) code forimproving system response.

Existing RAID (Redundant Array of Independent Disks) systems havemultiple pathways, or routes, for reading requested data. Often,however, there are only a few available pathways, of which only onepathway is efficient. For example, a RAID 5 system provides two pathwaysfor reading information. One pathway is by directly reading the sectorcontaining the requested information. The second pathway is byreconstructing the data sector containing the requested information byreading the appropriate sector from each other storage unit in the RAID5 array. As another example, an N storage unit array that is configuredas a RAID 6 system can read a data sector directly or can read all otherstorage units, except one. Accordingly, there are N ways for RAID 6 toread a data sector. In both instances, there is one efficient pathwayand one or more inefficient pathways.

Thus, when there are many pathways for obtaining requested data, forexample, more pathways than the number of storage devices in the array,it is not trivially clear which pathway provides in the highestperformance for a storage system. Consequently, what are needed are asystem and a method for determining which pathway to select when astorage system uses a redundancy method having many pathways torequested data.

BRIEF SUMMARY

The subject matter disclosed herein provides a system and a method fordetermining which pathway to select when a storage system uses aredundancy method having many pathways to requested data.

The advantages of the subject matter disclosed herein are provided by apathway determination system for a data storage system having N storagedevices and more than N pathways for retrieving requested data from thedata storage system. The subject matter disclosed herein also permits atleast one of the storage devices to be a failed storage device. Thepathway determination system includes a sorter, an assigner and acollector. The sorter receives a read request and separates the readrequest into an appropriate segment size for sending to the storagedevices of the data storage system. The assigner includes a permutationgenerator that generates a set of read permutations satisfying the readrequest, and a cost calculator that calculates an expense of eachpermutation based on a metric calculated from performance informationreceived from the storage devices of the storage system. The readpermutations can be generated after the read request is received.Alternatively, the read permutations can be precalculated before a readrequest is received, based on the architecture of the storage system.The cost calculator uses queue length information and estimated currentcost information to assign a value based on a metric to the readpermutations. Additional information on the costs are passed back to thepermutation generator as hints for the permutation generator to reducethe number of permutations it generates. The assigner selects a readpermutation from the set of read permutations. The selection of readpermutation is based on the value that the cost calculator assigned it.The metric used by the cost calculator is based on the anticipatedresource use of the permutation, such as on a current workload balancefor the storage devices of the data system, an estimated delay beforethe requested data can be retrieved from the storage devices of thestorage system, a number of outstanding requests in the queue of astorage device of the storage system, and/or a total queue for alloutstanding requests that have been received by the storage system. Theassigner sends the selected read permutation to the storage devices ofthe storage system. The collector receives the requested data from the Nstorage devices in response to the selected read permutation being sentto the storage devices. The metric can be dynamically changed based on achange in operating conditions of the storage system. Alternatively, themetric can be periodically changed based on operating conditions of thestorage system.

The subject matter disclosed herein also provides a method fordetermining a pathway for obtaining data stored in a data storage systemhaving N storage devices and more than N pathways for retrievingrequested data from the data storage system. The storage system caninclude at least one failed storage device. According to the subjectmatter disclosed herein, a read request is received and separated intoan appropriate segment and size for sending to the storage devices ofthe data storage system. A set of read permutations satisfying thereceived read request are generated either at the time the read requestis received or in advance. A read permutation is selected from a set ofread permutations based on a metric. The metric can be based on acurrent workload balance for the storage devices of the data system, anestimated delay before the requested data can be retrieved from thestorage devices of the storage system, a number of outstanding requestsin the queue of a storage device of the storage system, and/or a totalqueue for all outstanding requests that have been received by thestorage system. An expense for each permutation is calculated based onthe metric and based on performance information received from thestorage devices of the storage system. Queue length information andestimated current cost information is generated and used for generatinga reduced number of read permutations based on the queue lengthinformation and the estimated current cost information. The selectedread permutation is sent to the storage devices of the storage system.The requested data is received from the N storage devices in response tothe selected read permutation being sent to the storage devices, and thesatisfied read request is returned to the requester. The metric can bedynamically changed based on a change in operating conditions of thestorage system. Alternatively, the metric can be periodically changedbased on operating conditions of the storage system.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter disclosed herein is illustrated by way of example andnot by limitation in the accompanying figures in which like referencenumerals indicate similar elements and in which:

FIG. 1 shows a functional block diagram of an exemplary embodiment of asystem for determining the best pathway for obtaining requested databased on a defined metric according to the subject matter disclosedherein;

FIG. 2 is a functional block diagram showing greater detail of anassigner, shown in FIG. 1, according to the subject matter disclosedherein;

FIG. 3 depicts allowed permutations generated by a permutationgenerator, shown in FIG. 2, for an exemplary SOLD triple according tothe subject matter disclosed herein;

FIG. 4 is a functional block diagram showing greater detail of a costcalculator, shown in FIG. 2, according to the subject matter disclosedherein; and

FIG. 5 is a functional block diagram showing greater detail of acombiner, shown in FIG. 1, according to the subject matter disclosedherein.

DETAILED DESCRIPTION

The techniques of the subject matter disclosed herein are applicable toa RAID-type system in which there are more than N pathways for returninga requested data sector, such that N is the number of storage devices inthe system. Arbitrary read requests received from a host system areseparated into appropriate segments and sizes for dispatch to the datastorage devices of the system. The various pathways to the requesteddata are evaluated and the best permutation of the read request isselected and issued to the storage device array. After the selectedpermuted read has been gathered, the requested information is determinedand returned to the host controller.

The best pathway to the requested data is determined based on a metric,such as the current workload balance for the storage devices in thesystem, an estimated delay before the requested data can be retrieved,the number of outstanding requests in the queue of a storage device,and/or the total queue for all outstanding requests that have beenaccepted. The metric may be predetermined or can be continuallyreevaluated and dynamically changed, such as by retroactively alteringthe queues of the storage devices of the system, so that the bestpathway is selected as circumstances and operating conditions change.When one or more storage devices in the system are unavailable, thesubject matter disclosed herein selects the best pathway from theremaining available pathways for obtaining requested data. Informationregarding the metrics that are used for selecting the best pathway canbe made available externally to the storage system for evaluation by amaintenance entity.

FIG. 1 shows a functional block diagram of an exemplary embodiment of asystem 100 for determining the best pathway for obtaining requested databased on a defined metric according to the subject matter disclosedherein. System 100 is contained within a storage system (not shown)using a 3 data+3 parity MDS (maximum distance separation) erasure code.A primary aspect of a storage system is to process read requests from ahost controller (not shown), dispatch the commands to the storagedevices of the system, and then return the retrieved data to the hostcontroller. A 3+3 MDS erasure code configuration allows three diskfailures without loss of data; while providing that data from any sectoron a data disk can be reconstructed by combining the data contained inthe appropriate sector from any three of the other disks. Accordingly,the subject matter disclosed herein can be adapted to suit otherredundancy schemes, such as RAID 51, other product codes and larger MDScodes.

System 100 includes a sorter 106, an assigner 108, an array of storagedevices 110-120, and a combiner 122. Storage devices 110-120 will alsobe respectively referred to herein as disks A, B, C, which contain data,and P, Q and R, which contain parity information. While storage devices110-120 are illustratively shown in FIG. 1 as disk drives A, B, C, P, Qand R, respectively, other mass storage devices, such as Random AccessMemory (RAM) storage devices, optical storage device, and tape storagedevices, can be used as storage devices 110-120.

A read request 102 that is received from the storage system (not shown),in which system 100 is contained, is passed to sorter 106. Sorter 106sorts each read request 102 in a well-known manner into a Segment (thatis, a region of the array of storage devices 110-120 corresponding tothe block address of the requested data), an Offset within the segment,a Length, and a Disk (i.e., data disk A, B or C), collectively referredto herein by the acronym SOLD. Each SOLD is then passed from sorter 106to assigner 108. Assigner 108 selects the best pathway to the dataidentified by the SOLD. Assigner 108 also passes a determination forsatisfying a SOLD to combiner 122 as a state update information 124.Combiner 122 reconstructs the data identified by the SOLD from theresults received from storage devices 110-120, and a satisfied request126 is passed back to the storage system that is external to system 100.Combiner 122 also provides feedback information 128 to assigner 108.

FIG. 2 is a functional block diagram showing greater detail of assigner108, shown in FIG. 1. Assigner 108 includes SOLD queues 204-208 for thedata disks, a SOLD gatherer 210, a permutation generator 212, a costcalculator 216, a permutation multiplexer 218 and storage device queues220-230. Storage device requests 202 a-202 c for data disks A, B and Cthat are received from sorter 106 (FIG. 1) respectively enterrandom-access SOLD queues 204-208. SOLD queues 204-208 are configured aswell-known I/O queues, and may combine known Quality-of-Service (QoS)features, such as windowing and elevator sorting. SOLDs that are withinthe same segment are removed from queues 204-208 by a SOLD gatherer 210,which then combines the removed SOLD to form a map of requests forstorage devices 110, 112 and 114 (i.e., disks A, B and C) within thatsegment. The combined SOLDs that are within the same segment arereferred to herein as a “SOLD triple”. A permutation generator 212examines each SOLD triple and generates the allowed permutations foreach received read request that satisfies the SOLD triple. The generatedpermutations are passed to a cost calculator 216 that determines theexpense of each permutation based on a defined metric. The leastexpensive permutation is selected by permutation multiplexer 218 basedon the defined metric. The selected permutation is passed to queues220-230 for the storage devices 110-120, respectively. State updateinformation 124 about the selected permutation is passed forward frompermutation multiplexer 218 to combiner 122 (FIG. 1). Information 232a-232 f relating respectively to queue lengths and estimated currentcosts of queues 220-230 are passed back from storage device queues220-230 to cost calculator 216. The cost of a queue is the amount thatthe cost metric is increased by placing the chosen request on that queue(one of its relationships is to the length of the queue). Additionally,queue length and estimated current cost information 232 is filtered bycost calculator 216 and passed to permutation generator 212 as hintinginformation 234. Hinting information 234 is used by permutationgenerator 212 to reduce the number of permutations that must besupplied. Performance information from storage devices 110-120 (FIG. 1)is supplied as feedback information 128 to cost calculator 216 forrefining the costing analysis.

FIG. 3 depicts allowed permutations generated by permutation generator212 for an exemplary SOLD triple 302. SOLD triple 302 depicts a solitaryread request on disk A and no read requests on disks B and C.Permutation generator 212, in the absence of any hinting information234, produces all allowed permutations 310-330 over disks A, B, C, P, Qand R that satisfies SOLD triple 302. In particular, permutation 310represents a single disk A read 310. Additionally, all combinations ofthree-disk reads of the six available disks A, B, C, P, Q and R forreconstructing the request on disk A are shown. More specifically,permutation 312 represents disk reads of disks C, P and R. Permutation314 represents disk reads of disks B, C and Q. Permutation 316represents disk reads of disks B, P and R. Permutation 318 representsdisk reads of disks B, C and R. Permutation 320 represents disk reads ofdisks B, C and P. Permutation 322 represents disk reads of disks B, Qand R. Permutation 324 represents disk reads of disks C, P and Q.Permutation 326 represents disk reads of disks B, P and Q. Permutation328 represents disk reads of disks P, Q and R. Lastly, permutation 330represents disk reads of disks C, Q and R. Each request, 312-330, isable to reconstruct the SOLD requested 302 by way of the design of theparity calculation.

Hinting is used by permutation generator 212 for eliminating disk readcombinations. For example, in a situation in which hinting providesinformation that disks C, P and Q are preferred because disks A and Bare heavily loaded and disk R has failed, permutation generator 212would only generate permutations 310 and 324. The process of hintinginforms the permutation generator 212 which disks would be prohibitivelyexpensive to use.

FIG. 4 is a functional block diagram showing greater detail of costcalculator 216, shown in FIG. 2. Cost calculator 216 includes fixed diskcost multipliers 408-418, weighting multipliers 420-430, and a costsummer 432. Permutations 310-330 are respectively input to fixed diskcost multipliers 408-418. The value of each respective fixed diskmultiplier is adjusted based on the disk's measured performance throughfeedback 128. The output of each respective fixed disk multiplier isweighted by the length of each corresponding disk queue for disks A, B,C, P, Q and R through 232 a-f. The result for each permutation iscalculated by summer 432 and output as permutation expense 436. Thesummer 432 may add the input results, or may expense the largest inputresult, or may use another appropriate algorithm. If necessary, hintinginformation 234 is output to permutation generator 212 (FIGS. 2 and 3).

FIG. 5 is a functional block diagram showing greater detail of combiner122, shown in FIG. 1. Combiner 122 includes disk return queues 504-514,a permutation return buffer 516, a permutation collector 518, an actualcost calculator 520 and a reconstructed data return buffer 522. Eachdisk A, B, C, P, Q and R respectively returns data to disk return queues504-514, which operate in a well-known manner. The selected permutationfor a SOLD triple is output from permutation multiplexer 218 topermutation return buffer 516 as state update information 124.Permutation collector 518 receives a completed permutation in diskreturn queues 504-514 and the selected permutation from permutationreturn buffer 516. The actual cost of the permutation is then calculatedby actual cost calculator 520. Actual cost information is returned tocost calculator 216 as feedback information 128 (FIGS. 2 and 3). Thedata for the original SOLD is reconstructed by reconstructed data returnbuffer 522 and output as return data 126 (FIG. 1).

For example, consider the costing of the situation wherein which disks Aand B are moderately loaded and disk R has failed; and in which the SOLDrequests one sector from disk A. Further, the metric in this exampleuses a trivial estimate of the sum of queue times for retrieving thedata as the cost. Assume that disk A queue 220 has a length of 40outstanding requests, disk B queue 222 has a length of 35, disk C queue224 has a length of 5, disk P queue 226 has a length of 8, disk Q queue228 has a length of 7 and disk R queue 230 has an indeterminate length.These queue lengths are passed to the cost calculator 216 throughpathways 232 a-f. Cost calculator 216 knows that requests to identicaldisks A, B, C, P and Q take 10 ms to complete and these weights arestored in fixed disk cost multipliers 408-416. Disk R, having failed, istaking more than 10⁶s to complete its requests 418. Hinting pathway 234informs the permutation generator 212 that disk R has failed. Thus, whenthe SOLD request for disk A is received 302, the permutation generator212 generates pathway permutations on disk A 310, disks BCQ 314, disksBCP 320, disks CPQ 324 and disks BPQ 326. The metric for pathway 310 is40×10 ms+0+0+0+0+0=400 ms, and for pathway 314 the metric is 0+35×10ms+5×10 ms+0+7×10 ms=470 ms. Similarly, pathway 320 has a summed cost of480 ms, pathway 324 has a cost of 200 ms and pathway 326 has a cost of500 ms. Thus, the lowest cost path is pathway 324 issuing reads to disksC, P and Q. The chosen pathway is conveyed to combiner 122 through stateupdate 124. Combiner 122 receives the relevant data from disks C, P andQ for reconstructing disk A′s information in the permutation collector518. The measured times to complete the request is collated in 520 andthe results used to update cost calculator 216 through feedback 128. Inthis example, the metric is very simple and the working disks areidentical in performance so feedback is not required. A moresophisticated cost calculator, however, may, for example, account forthe fact that the time for a disk to complete a request is not linear inthe queue length, as is assumed here. The feedback mechanism permits thecost calculator to use a more accurate estimate.

The subject matter disclosed herein can determine an optimal pathway forretrieving data from a data storage system as requests for data arereceived. Alternatively, the subject matter disclosed herein determinesoptimal pathways in advance, based on current operating conditions ofthe storage system, and selects a predetermined optimal pathway for eachreceived data request. Moreover, the subject matter disclosed herein canbe configured as a processing system within a storage system thatexecutes machine-language instructions or as individual componentscontained within a storage system that perform the respective functionsof the subject matter disclosed herein.

While the subject matter disclosed herein has been described withrespect to a specific example illustrating a general process forselecting a best pathway to data and for improving performance of datareads of a storage system, those skilled in the art will appreciate thatthere are numerous variations and permutations of the above describedsystems and techniques that fall within the spirit and scope of thesubject matter as set forth in the appended claims.

1. A pathway determination system for a data storage system comprising Nstorage devices and more than N pathways for retrieving requested datafrom the data storage system, the pathway determination systemcomprising: a sorter capable of receiving a read request and separatingthe read request into an appropriate segment size for sending to thestorage devices of the data storage system; an assigner capable ofselecting a read permutation satisfying the received read request, theselected read permutation being based at least in part on apredetermined metric, and the assigner being capable of sending theselected read permutation to the storage devices of the storage system;and a collector capable of receiving the requested data from the Nstorage devices in response to the selected read permutation being sentto the storage devices.
 2. The system according to claim 1, wherein theassigner is further capable of generating the read permutationssatisfying the received read request.
 3. The system according to claim2, wherein the assigner generates the read permutations before the readrequest is received.
 4. The system according to claim 1, wherein theassigner comprises: a permutation generator capable of generating theread permutations; and a cost calculator capable of calculating anexpense of each permutation based on the predetermined metric.
 5. Thesystem according to claim 4, wherein the cost calculator is capable ofutilizing queue length information and estimated current costinformation, and wherein the permutation generator is capable ofgenerating a reduced number of read permutations based at least in parton the queue length information and the estimated current costinformation.
 6. The system according to claim 5, wherein the costcalculator is capable of calculating the expense of each permutationfurther based at least in part on performance information received fromthe storage devices of the storage system.
 7. The system according toclaim 1, wherein the storage system comprises at least one failedstorage device.
 8. The system according to claim 1, wherein the metricis dynamically changed based at least in part on a change in operatingconditions of the storage system.
 9. The system according to claim 8,wherein the metric is periodically changed based at least in part onoperating conditions of the storage system.
 10. The system according toclaim 1, wherein the metric is based at least in part on a currentworkload balance for the storage devices of the data system.
 11. Thesystem according to claim 1, wherein the metric is based at least inpart on an estimated delay before the requested data can be retrievedfrom the storage devices of the storage system.
 12. The system accordingto claim 1, wherein the metric is based at least in part on a number ofoutstanding requests in the queue of a storage device of the storagesystem.
 13. The system according to claim 1, wherein the metric is basedat least in part on a total queue for all outstanding requests that havebeen received by the storage system.
 14. A method for determining apathway for obtaining data stored in a data storage system comprising Nstorage devices and more than N pathways for retrieving requested datafrom the data storage system, the method comprising: receiving a readrequest from a requester; separating the read request into anappropriate segment and size for sending the storage devices of the datastorage system; selecting a read permutation from possible readpermutations satisfying the received read request; sending the selectedread permutation to the storage devices of the storage system; receivingthe requested data from the N storage devices in response to theselected read permutation being sent to the storage devices; andreturning the satisfied read request to the requester.
 15. The methodaccording to claim 14, further comprising generating the readpermutations satisfying the received read request.
 16. The methodaccording to claim 15, wherein generating the read permutations isperformed before the read request is received.
 17. The method accordingto claim 15, further comprising calculating an expense of eachpermutation based at least in part on the predetermined metric.
 18. Themethod according to claim 17, further comprising: generating queuelength information and estimated current cost information, andgenerating a reduced number of read permutations based at least in parton the queue length information and the estimated current costinformation.
 19. The method according to claim 18, wherein calculatingthe expense of each permutation is further based at least in part onperformance information received from the storage devices of the storagesystem.
 20. The method according to claim 14, wherein the storage systemcomprises at least one failed storage device.
 21. The method accordingto claim 14, further comprising dynamically changing the metric based atleast in part on a change in operating conditions of the storage system.22. The method according to claim 14, further comprising periodicallychanging the metric based at least in part on operating conditions ofthe storage system.
 23. The method according to claim 14, wherein themetric is based at least in part on a current workload balance for thestorage devices of the data system.
 24. The method according to claim14, wherein the metric is based at least in part on an estimated delaybefore the requested data can be retrieved from the storage devices ofthe storage system.
 25. The method according to claim 14, wherein themetric is based at least in part on a number of outstanding requests inthe queue of a storage device of the storage system.
 26. The methodaccording to claim 14, wherein the metric is based at least in part on atotal queue for all outstanding requests that have been received by thestorage system.